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This  report  is  part  of  a series  of  reports  on 
operations  useful  for  Automatic  Submarine  Detection 
and  Classification.  Other  reports  in  the  series  may 
be  obtained  by  writing  Dr.  Edward  A.  Patrick,  Associate 
Professor  of  Electrical  Engineering,  Purdue  University. 


Abstract 


The  application  of  the  Karhunen-Loeve  expansion  to  signal  analysis  problems 


involves  eigenvector  .and  eigenvalue  calculations.  Practical  questions  which 


arise  are  how  many  eigenvectors  should  one  calculate  and  how  many  sample  wave' 


forms  must  one  obtain  to  determine  these  eigenvectors  with  a specified  accuracy 


Hie  answers  to  these  engineering  questions  are  the  concern  of  this  paper 


In  addition,  a rapid  algorithm  for  numerically  calculating  the  eigenvectors 


and  eigenvalues  of  an  autocorrelation  matrix  is  presented  which  is  applicable 


mainly  to  signal  analysis  problems 


1 . Introduction 


The  Kferhunen-Loeve  expansion  is  well  established  as  a means  of  representing 


a random  process  in  signal  analysis.  Hie  application  of  the  expansion  is 
conceptually  straightforward.  We  obtain  N sample  waveforms  x^t),  k=l, . . . ,N, 
tf[0,T],  and  time  sample  each  at  n points,  t^,  A=l, ...,n,  t^tlO^T].  We  then 
calculate  the  autocorrelation  matrix,  S,  whose  element,  s.  , is 


Denote  the  ith  element  of  the  ith  eigenvector  of  S by 
Karhunen-Loeve  expansion  for  x.  (t.)  is 


Although  the  expansion  of  (2)  is  an  exact  representation  of  XjJt^J's  with  n 
eigenvectors,  an  approximation  can  be  obtained  by  selecting  a smaller  number 
of  eigenvectors.  The  mean  square  error  of  the  approximation  is  the  summation 
of  the  eigenvalues,  X. 'b,  whose  eigenvectors  ere  not  selected. 


Thus,  the  central  problem  in  the  Karhunen-Loeve  expansions  is  to  calculate 


a he 
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the  eigenvectors  and  eigenvalues  and  to  select  the  eigenvectors  with  the  larger 
eigenvalues  as  the  important  basis  functions  to  represent  a given  process. 
Although  this  procedure  is  straightforward,  we  face  engineering  problems 
especially  when  waveforms  are  analyzed,  mainly  because  the  number  of  sampling 
points,  n,  becomes  very  large  in  signal  analysis. 

This  paper  is  concerned  with  solving  those  engineering  problems  such  is 
(l)  how  many  samples,  N,  are  needed  to  estimate  the  eigenvalues  end 
eigenvectors  with  a specified  accuracy. 


i 1 
II 


(2)  how  many  sampling  points,  n,  should  be  selected  to  characterize  the 
process  properly. 

The  quick  evaluation  of  N and  n is  appreciated  especially  when  n is  large 
such  that  many  eigenvalues  end  eigenvectors  should  be  calculated. 

I 

In  addition  to  the  above,  this  paper  offers  a program  for  rapid  eigenvector- 
eigenvalue  calculation  which  has  been  developed  in  relation  with  the  above  study. 

The  basic  approach  of  this  program  is  to  celculate  2n  dimensional  eigenvalues  and 
eigenvectors  from  n dimensional  eigenvalues  and  eigenvectors,  as  the  ffest 
Fourier  Transform  does  for  Fourier  transforms.  The  computation  time  of  the 

: 

program  is  about  three  times  faster  then  conventional  programs. 


Eigenvectors  and  eigenvalues  of  a 


In  this  section  we  derive  first  order  approximations  for  the  eigenvector 
and  eigenvalues  of  a perturbed  matrix  in  terms  of  those  of  the  non-perturbed 
matrix.  These  approximations  are  used  in  the  analysis  which  follows. 

Let  SQ  be  a real,  symmetric  (nxn)  matrix  end  let  dS  he  a reel,  symmetric 
perturbation  matrix.  Let  0i  and  X^  i=l, ...,n,  he  the  eigenvectors  and  eigen 
values,  respectively,  of  SQ.  Assume  that  the  X^s  are  distinct.  We  wish  to 
obtain  a first  order  approximation  of  the  eigenvectors  and  eigenvalues  of  S, 
in  terms  of  0i's  and  X^'s  where 

S = S + dS  (5 

o w 

These  may  he  obtained  by  retaining  the  terras  of  first  order  or  lower  of  the 


equation 


where 


(SQ  + dS)(0i  + d0£)  - (Xt  + dXi)(0i  + d0i)J 


So*i  = Xi^i* 

The  resulting  equation  is 

SQd0i  + dS01  - Xid0i  + dX10i. 

To  calculate  dX^,  we  premultiply  equetion  (8)  by  0^  and,  since  0^SQ  = 
X^  and  * 6^^,  we  have 

dXx  = 0^dS0i. 

We  can  write  <30^  as  a linear  cooibination  of  the  0j*s  as  follows: 

*!  ■ A *uV 


where 


If  we  premultiply  equation  (8)  by  0?  and  rearrange,  we  have  for  I^J 


To  determine  b..  we  impose  8 first  order  normalization  condition  on  0,  + d0. 


0 for  l^j,  we  summarize  this  section  as 


Noting 


Estimation  of  eigenvectors  and  eigenvalues 


The  estimation  of  eigenvectors  and  eigenvalues  is  made  from  the  sample 


autocorrelation  matrix,  S.  Therefore,  the  estimator  output  is  a random  variable 


with  its  mean  and  variance.  The  variance  is  very  much  a function  of  the  number 


of  samples,  and  approaches  zero  with  infinite  number  of  samples.  In  thi 


section 


the  variances  of  estimated  eigenvalues  and  eigenvectors  will  be  discussed.  The 


purpose  of  this  discussion  is  to  find  the  necessary  number  of  samples  to  achieve 


a specified  accuracy  for  the  estimation  of  eigenvectors  and  eigenvalues 


The  autocorrelation  matrix,  S 


where  X is  a n dimensional  sample  vector 


In  practice,  however,  S is  not  known  a priori.  Rather,  we  are  given  a 


set  of  N sample  vectors,  X^, ...,X^ 
the  sample  autocorrelation  matrix 


which  we  use,  as  follows,  to  calculate 


Let  0^  and  X^,  1=1, ...,n,  be  the  eigenvectors  and  eigenvalues  respectively 
of  S and  similarly  define  0.  end  for  S.  Just  as  S is  an  estimate  of  S, 


The  statistics  of  the  eigenvectors  and  eigenvalues  of  a matrix  of  random 


variables  have  been  studied  previously .[l] , [2]  The  general  approach  is  to 
calculate  the  distribution  of  S and  from  this  find  the  distribution  of  the 


However,  since  S = S for  N sufficiently  large,  we  may  use  the  approximations 
(li>)  and  (16)  to  express  0.  and  X.,  i.e.. 


; I 

I § 


where 


i=l, • • • y n. 


r 


l 0 


i-J  . 


Hrst,  we  consider  the  mean  value  of  the  estimates.  Prom  (17)  and  (18), 
E{j2^S0j}  = 0^E{S}0. 


= 0>i 

= Xi6ij  * 


It  follows  from  (19) > (20)  and  (21)  that 

= 0±, 


E{Xi}  = Xi.  (2* 

A 

Thus  the  estimates  are  seen  to  be  unbiased.  The  mean  square  errors  of  X^  end 
0^  are  given  by 

1 * - X*)2}  = B{X2}  - X2 

xi 

- E{(0^)2}  - x2  , (2; 

02  = Ef||0  - 0.1 12}  = I E{b2  } 

0 1 1 J=1 

n Ef(0^§0  )2} 

~ I i-J-*-  . (26 

j=i  (x.  - x.r 

J^i  J 

The  expectations  of  (25)  end  (26)  can  be  obtained  from  the  moment  generating 


-7‘ 


functions  of  the  samples,  as  shown  in  the  Appendix. 

(A)  Gaussian  Cases 

For  the  special  case  where  the  X^'s  come  from  a gaussien  distribution 
with  mean  vector  M and  autocorrelation  matrix  S,  (25)  and  (26)  become,  from 
(ALO)  of  the  Appendix, 

°f  (27) 


2 ~ 1 


n X X - 2^  2 

cT  — y — . mmn.M  m 

0±  N j=l  (x±  - X f 
jA  J 


(23) 


where 


= 


(i=l,2, . . .,n).  (29) 

Equations  (27)  and  (28)  indicate  that  mean  square  estimation  error  is  the  product 
of  l/N  and  some  coefficient  which  is  independent  of  N.  These  coefficients  are 
determined  by  S and  M. 

2 

Much  simpler  expressions  are  available  as  upper  bounds  of  a.  and 

2 ^i 

<rA  by  dropping  out  the  (j,  terms  as  follows: 


£<  ^2.2  £ 
f - N Xi  °r  'Xi  - » 9 


(30) 


1 n X^X  . 

af  E “ 

0,  N j=l  (X±  - Xj) 


2 N 


i ? 


il^i  _4i 


jA 


J=1  (1  - X ,/X . ) 

jA  J 1 


2 _ N ■ i* 


(31) 


The  equalities  of  (30)  and  (31)  hold  when  ^ = 0 for  ell  i's. 

OO  . 

Equation  (30)  shows  that  o^  /\.  is  simply  bounded  by  2/N  regardless  of  i 

xi  . 

and  n.  On  the  other  hand,  from  (31),  o*  depends  on  i end  n as  well  as  l/N. 


0, 


yj  of  (jl)  is  a function  of  n-1  Xj/X^s  (j=l,2, ..  .,n:  j/i).  However,  yi 


is 


roughly  determined  by  a single  or  a few  terms  whose  X./X. 's  are  close  to  one 
This  property  will  be  discussed  in  detail  for  the  following  example . 

(b)  Example:  stationary  process 


I«t  x(t),  t€[0,T],  be  a stationary,  Gaussian  random  process  with 
R(t)  = E{x(t)x(t-T)} 


= exp{-a|r^. 


If  x(t)  is  time  sampled  at  t = — , Ji=l, . . .,n,  the  S becomes  a matrix  whose 
element,  s.  , is 


where 


sAm  = e*P{-a|MT/n}, 

|4-ml/n 
= P ' t 

H , m=l  y • • • y n y 


p = exp{-Qff} . 


Suppose  that  the  product  off  is  such  that  p « 1.  Then  the  eigenvalues  of  S 
are  given  approximately  by 

X±  = K t(i/T),  ( 
where  ^ ( * ) is  the  spectral  density  of  x(t)  and  K is  a constant.  Then,  since 
the  spectral  density  of  (32)  is  ■■ • • , the  ratios  of  adjacent  eigenvalues 


a2  +02 


are  approximately 


i + 1 


Xi  a2  + (i  + 1 )Z/T? 


c?  + i2/^ 


= 1 - 


(l  AZQ/oftr8 

1 -i-  (i  + l)2/a?T2 


These  ratios  are  independent  of  n and  approach  1 as  i becomes  large.  It  can 

2 

be  shown,  also,  that  for  large  1,  q*  i6  independent  of  o?T,  and  hence  p,  and 

% 

2 A 
varies  essentially  as  i . 


for  the  matrix 


ore  plotted  as  a function  of  n in  Figure  1.  We  see  that  each  y.  becomes  constant 


as  n increases  beyond  i.  Figure  2 shows  these  steady  state  values  of  y.  as  a 


For  i > Ij-  we  see  that 


function  of  i for  various  values  of 


linearly  with  i on  the  logarithmic  plot  and  is  fairly  insensitive  to  p.  This 


result  is  consistent  with  our  approximate  theory 


The  number  of  samples  required  so  as  to  estimate  the  eigenvectors  accurately 


depends  on  the  number  of  eigenvectors  and  hence  the  dimension.  In  this  example 


for  a given  accuracy  the  number  of  samples,  N,  required  increases  as  n‘ 
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4.  Determining  the  nuraber  of  eigenvectors  to  calculate 


In  this  section,  discussion  will  be  given  for  determining  the  sampling 
interval  or  how  many  eigenvectors  should  be  calculated  for  a process  in  a given 
time  region. 

Figure  3(a)  shows  a waveform  x(t).  If  we  sample  the  waveform  at  four 
sampling  points  tg,  t^,  tg  and  tg,  we  have  subsequently  four  eigenvalues  as 
shown  in  Figure  3(b).  Doubling  the  sampling  points  from  four  to  eight  as  shown 
by  the  dotted  lines  in  figure  3(a)>  we  change  the  previous  four  eigenvalues 
and  add  another  four  eigenvalues  as  shown  by  the  dotted  lines  in  Figure  3(b). 

If  four  sampling  points  are  enough,  the  doubling  of  the  sampling  points  must 
generate  only  very  small  new  eigenvalues. 

Tims,  the  basic  approach  of  this  paper  to  check  whether  we  have  enough 
sampling  points  or  not  is  to  calculate  the  summation  of  the  eigenvalues  which 
are  generated  by  doubling  the  sampling  points. 

Suppose  the  process  is  sampled  at  n points  in  [0,T] . Then  the  wave  vector 


x(t2) 


x°  = X<V 


lx(t2n>J 

and  the  autocorrelation  matrix  is 

Sn  = E[xY?] 

■ Mx(t21)x(t2J))] 

• wii’V1  > 


where  [a  ] is  a matrix  whose  ij  element  is  a . . If  the  sampling  rate  ic 

tJ  1 J 

doubled,  the  wave  vector  and  autocorrelation  matrix  become  respectively 
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f:l 


I . 


x2" 


x(t2) 


x(t2n) 


x^) 


-G&. 


(39) 


ond 


= Efx2,V5nT3 

f cjn  ,”1 

sn  si2 

sn  sn 

L 21  22) 

m 


where  is  the  same  as  Sn  of  (38)*  The  order  of  x(t^)'s  is  given  as  (39) 
only  for  simplifying  discussion.  Replacing  the  order  of  x(t  )'s  as  x(t..  ), 

/j 


x(t,,),...  does  not  change  the  essential  points  of  the  discussion. 


If  n is  large,  then  S^,  S2l  and  of  (*40)  all  become  close  to 


22 


„2n 


each  other.  Then,  S can  be  approximated  by 


,2n 


Sn  Sn 
S11  S11 


11  °11  ) 


Assume  that  the  eigenvectors  of  S^,  0?,  have  been  calculated.  The  eigenvector 


,2n 


matrix  and  the  eigenvalue  matrix  of  Sq  are  then 
.2n 


r 


1 

72 


*n  *n 


*n  -$n 


(42) 


.2n 


T2An  01 

L 0 Oj  » 


(43) 


I 


where 
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$ = [f^  •••  • 


(14;) 


It/  (15),  the  eigenvalues  of  S can  be  approximately  evaluated  by  using 


the  eigenvectors  of  Sq*1,  $^n.  These  are  the  diagonal  terras  of 


~2n  ~ _2n  JZn  ,2n 

Lr  = $ O 5> 


1 

2 


s'  1 

.n 

.n 

l"_n  _n  > 

f n 

n h 

$ 

T 

$ 

T 

• 

S11 

12 

• 

$ 

§ 

A11 

k* 

-*n 

> 

sn 

lS2l 

sn 
22  ) 

*n 

-5  ) 

'rn 

G11 


,n 


12 


,n 


,n 


21 


22 


(*5) 


where 


"ll 


1 »n  f<sn  J.  Cn  A on  ^ Qn  \*n 

= - i (s71  + slft  + s91  + S„)$  , 


'll 


12 


21 


- \ *"  <SU  - ^ * S21  - S22>*“" 


22  ‘ 

,n  ,,n 


21 


1 n / n n n _n  x,n 

2 $ 'Sn  + Si9  “ Soi  “ SooJ*  > 


T 


'11 


12 


21 


22' 


i $n(s”  - S?  - S*  + S*  )in. 


(4G) 

(^7) 

(i;3) 

(*9) 


22  2 v“ll  12  21  ' “22- 

If  S^,,  and  are  all  close  to  S^,  of  (46)  becomes  close  to  2An,  end 
Gg2  of  (49)  becomes  close  to  0.  Thus,  the  diagonal  terms  of  ere  the  n 
eigenvalues  which  we  had  for  n sampling  points  and  are  modified  by  doubling 
the  sampling  points.  On  the  other  hand,  the  diagonal  terras  of  G^p  are  the 
new  eigenvalues  which  are  generated  by  doubling  the  sampling  points. 

Therefore,  the  summation  of  the  new  eigenvalues  is  calculated  by 

tr  °22  ' £ X2i-1  ' I tr<SU  - S“l2  - S21  + S22>- 


(50) 


T 

where  $ can  be  eliminated  from  the  calculation  of  trace  because  ? $ x 


in  *n 


I. 


£ X2i-1  iS  "0r“li“d  * = <\>i  * ^21-1 )' 


3 -TT 


^U<2< 


iJL^X2i+X2i-l^  tr^Gll+G22^  2tr^Sll+S22^ 


Or,  substituting  (40)  into  (51), 


n 1 iSi^R^tzi,t2i^’R^t21,t2i-l^"R^t2i-l,tZi^+R^t2i-l*t2i-l^ 

P = — ■ ■ ■ ■ 

If  x(t)  is  a stationary  process  and  if  tk  = |^,then  equation  (52)  reduces 


B(0)  - H(fe) 

P = • (53! 

2R(0) 

Equation  (52)  or  (53)  may  be  used  to  determine  n by  requiring  pn  < € where  f 
is  fixed  by  the  analyst. 

As  an  example,  suppose  R(t)  = e”^,  T=l,  and  we  want  to  choose  n so  that 
pn  < 0.05.  Ry  equation  (53) 


An  1 - e 

P " T 


and  so  we  should  calculate  n=5  eigenvectors. 


nmmewp 
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5*  Rapid  eigenvector-eigenvalue  calculation 

In  this  section,  a fast  program  is  offered  to  calculate  eigenvectors  end 
eigenvalues  in  waveform  analysis.  For  convenience,  we  shall  refer  to  e program 
which  calculates  eigenvectors  and  eigenvalues  as  an  EIGEN  program. 

Equations  (42)  and  (43)  indicate  that  2n  dimensional  eigenvectors  and 
eigenvalues  can  be  approximated  by  using  n dimensional  eigenvectors  and  eigen- 
values, particularly  in  signal  analysis  where  S 22,  S12  and  S2l  of  (40)  become 
close  to  S.p  with  large  n.  This  is  very  much  analogous  to  the  Fast  Fourier 
Transform,  where  2n  frequency  points  can  be  exactly  calculated  from  n frequency 
points . 

On  the  other  hand,  in  most  EIGEN  programs,  on  iterative  procedure  is  used, 
starting  and  forming  8 sequence,  such  that 

* = lim  , (tt) 

n — <*> 

which  satisfies 

«TS$  = A . (56) 

Furthermore,  since  a general  purpose  program  can  assume  little  about  the  structure 
of  S,  is  usually  taken  to  be  I.  Naturally,  if  we  select  which  is 

close  to  $ as  the  starting  point,  we  may  reduce  the  number  of  iterations. 

'Bie  basic  idea  of  the  Fast  EIGEN  program  of  this  psper  is  to  combine 
the  above  two  facts.  'Hist  is, 

(1)  approximate  2n  dimensional  eigenvectors  and  eigenvalues  by  n dimensional 
eigenvectors  and  eigenvalues,  and 

(2)  use  the  approximation  as 

In  most  cases  where  the  computation  time  becomes  serious  problem,  n is 


! 


large  so  that  the  above  approximation  becomes  better. 

A better  approximation  of  the  2n  dimensional  eigenvectors  than  (42)  is 


used  for  the  Jtest  EIGEN  program.  Uiis  is 


where 


n n n 
Sll*l  " *1  A1 

Sn  $n  = $n  A 
22*2  *2  c 


It  is  easily  seen  that  ^ is  an  orthogonal  matrix.  Then, 


»2nVn 


I (A^W)  i(A1-A2-B.BT) 
\ (A1-A2+B-BT)  |(A1+A2-B-BT) 


(57) 

(58) 

(59) 

(60) 


where 

B - • (61) 

Hie  approximation  of  (57)  has  following  two  advantages. 

(1)  Using  two  sets  of  n dimensional  eigenvectors,  and  $£,  the  phase 
lag  information  between  x(t^) -wave forms  and  x(t2_^  ^)  waveforms  is  taken  into 
account . 

T 

(2)  In  order  to  calculate  $2n  s2n$2n  of  (60)  only  n dimensional  matrix 
multiplication  as  (6l)  is  required.  Hie  number  of  multiplications  for  this 
calculation  is  roughly  estimated  as  2n^. 


When  Sjj,  S^,,  and  S22  are  all  close  to  each  other  with  sufficiently 
large  n,  then  and  «2,  and  A^>  Ag  and  B are  also  close  to  each  other.  Hiere 

p— 

fore,  $ S$  of  (60)  is  very  close  to  a diagonal  matrix.  Starting  from  this 
2n 

matrix  and  $ of  (57^  0 convensional  EIGEN  program  such  as  J8cobi'3  method 
can  start  to  reduce  the  off-diagonal  terms  further  [3]. 

Figure  4 shows  how  the  Fast  EIGEN  program  calculates  2m  dimensional 
eigenvectors  and  eigenvalues.  Assuming  2m=8  for  simple  explanation,  the 


procedure  is  as  follows: 


- ” 11 


; 


(1)  Provide  eight  one  dimensional  eigenvectors,  ell  1,  and  eigenvalues,  s ^ 

U=l,2, 

(2)  Approximate  four  two  dimensional  eigenvectors  by  (57)>  end  calculate 
(6o)  for  ell  four  cases.  Combination  should  be  made  as  (t^,  t^),  (t2,  t^), 

(t3,  t?)  and  (t^,  tQ). 

(3)  Use  the  results  of  (60)  and  (57)  as  "the  initial  condition  and  apply 
the  Jacobi's  method.  Pour  two  dimensional  eigenvectors  and  eigenvalues  are 
obtained. 

(4)  Repeat  (2)  and  (3),  increasing  the  dimension  from  two  to  four,  and 
from  four  to  eight.  The  number  of  sets  of  eigenvectors  end  eigenvalues  are 
reduced  from  four  to  two,  and  from  two  to  one . 

Hie  computation  time  is  roughly  estimated  bs  follows:  In  the  Jacobi's 

method,  it  has  been  reported  that  the  computation  time,  TJf  is 

Tj  = 10  u n3,  (62) 

where  u is  a multiplication  time  and  n is  the  dimension.  Assuming  that 
two  sets  of  n/2  eigenvectors  and  eigenvalues  are  available,  the  computation 
time  of  one  set  of  n dimensional  eigenvectors  and  eigenvalues,  Tn,  is 

Tn  = 6 Tj  = it)  0 u n3,  (63) 

where  Q depends  on  the  initial  approximation.  For  simplicity,  let  us  assume 
that  0 is  constant.  Hien,  the  computation  time  of  the  Fast  EIGEN  program,  T^, 


2°T2m  + Z^m-l  + ...  + 2mrlT2l 


t ; 

1 , I 


10  ©u  23m(l  + 2"2  + 2-4  + ...  + 

10  6 u 40(lOu  23"1) 

1-2  J 


4 

3 


^ 


Thus,  T^/Tj  is  almost  determined  by  0. 

For  the  Jacobi’s  method, 0 is  determined  experimentally.  The  computation 
time  is  essentially  proportional  to  the  number  of  Jacobi  iterations  required 
to  achieve  convergence  to  the  desired  accuracy.  Tlius  by  (63)  G is  given  by 


n 


(65) 


n/S  J i/? 

where  In  is  the  number  of  iterations  required  to  obtain  $ from  and 
and  Ij  is  the  number  of  iterations  required  conventionally.'1' 


Figure  5 shows  curves  of  In  and  Ij  for  a matrix  S whose  i,.j  element  i 

|i-j| 


s. . = v' 
ij 


m 


versua  the  matrix  dimension  for  various  values  of  v.  The  spacing  between  the 
Fast  EIGEN  and  conventional  curves  on  the  semi-log  plot  is  nearly  constant  Tor 
fixed  v,  and  0 varies  between  about  0.2^  and  O.35. 

Thus  computation  time  can  be  reduced  by  the  Fast  EIGEN  approach.  Hie 


significance  of  the  time  saving  of  course  depends  on  the  details  of  the 
application. 
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6.  Summary 

We  have  solved  in  part  some  of  the  engineering  problems  involved  with  the 
application  of  the  Karhunen-Loeve  expansion.  In  particular  we  have  outlined 
procedures  for: 

1.  determining  the  number  of  time  samples  to  take  from  the  waveform, 

2.  determining  the  number  of  sample  waveforms  required  to  accurately 
determine  the  eigenvectors,  and 

3.  reducing  the  computational  effort  in  numerically  determining  the 
eigenvectors . 

These  procedures  in  conjunction  with  a basic  understanding  of  the  Karhunen- 
Loeve  expansion  allow  its  straightforward  implementation  in  communication  and 
pattern  recognition  systems . 


k 
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Appendix 


l 


I > 


Calculation  of  Ef0^S0j)2] 

Recall  that  S is  given  by 

v£- 


k=l 


and  thus  we  have 


■ I j, 


k=l 


(A.l) 


(A.  ?.) 


where 


(A. 3) 


If  both  sides  of  (A. 2)  are  squared  and  expectation  is  taken,  the  result  is 

E{(0^0J2}  = ^2  E E 

1 J IT  k=l  k'=l  1 J 1 J 


N N 


= ^ E E e(yM}  E[yJVJ'} 

I T k=l  k'=l  J 1 J 
k/k' 

+ 7 r E{Y?  Y*  } , 

r k=i  1 J 


(A.4) 


since  the  X^'s  ore  independent. 

Now 

E{Y^]  . ^ EOtkX^}0J 

= ^=1  y • • •)  N • 

and  (A. 4)  may  be  re-written  as 


Sf(^J2l  - ^ Xf6  . 1 EOfVh  . 


(A. 5) 


(A. 6) 


'y  ’ N "iwij  N “l‘i‘JJ 
The  oupercript  on  Y is  dropped  since  the  X^’s  are  identically  distributed.  'Ibe 
last  term  In  (A ,6j  may  be  calculated  if  the  moment  generating  function  of  X is 
known.  'Hie  joint  moment  generating  function  of  Y^  and  Y^.  is 
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Mi  j (ti,^2  ) = Efexpft^X  + t20^X)} 

= Efexp  [(t^  + t^F.)X]] 

- Wi + Vj)'  ^ 

where  .)  is  the  moment  generating  function  of  X.  It  follows  then  that 


3M^,t2) 


E{Y^}  = 


Stl5t2 


a m (tl,tz) 

T 


i-Aj 


tl=t2=° 


(A.  8) 


at; 


i=j 


ti^2=o 


If  X comes  from  a multivariate  normal  distribution  with  mean  vector  M 


and  autocorrelation  matrix  £*  then 

EfY^)  , 2x25i  . + XiXj  . 2ll^ 

Combining  (A. 9)  and  (A .6)  we  have 


- #y  - s<^8u  + hh  - • 


2 2, 


■ 


where  is  given  by  (29)  • 


(A.  0) 


(A.  10) 


-21- 


List  of  References 

1.  T.W.  Anderson,  An  Introduction  to  Multivariate  Statistical  Analysis,  New 

York:  John  Wiley  end  Sons,  Inc.,  1958,  pp.  307-325. 

2.  M.A.  Girshick,  "On  the  Sampling  theory  of  roots  of  determinentel  equetions", 
Anne Is  of  Mathematical  Statistics.  Vol.  10,  1939,  pp.  203-224. 

3.  J.  Greenstadt,  "The  determination  of  the  characteristic  roots  of  a matrix 

by  the  Jacobi  method.  Mathematical  Methods  for  Digital  Computers.  A.  Ralston 
and  II.  S.  Wilf,  ed..  New  York:  John  Wiley  and  Sons,  Inc.,  1962,  pp.  84-9-1. 

4.  P.A . White,  "The  computation  of  eigenvalues  and  eigenvectors  of  o matrix", 

J.  Soc.  Indust.  Appl . Math.,  Vol.  6,  1958,  pp.  393-437. 


V 


Number  of  Iterations,  I . and  I 


